fix(cli): the metadata rubric records a linter crash instead of scoring it 100 / A - #15881
Conversation
…ing it clean `scoreMetadata` caught a `lintConfig` throw and continued with `issues = []`, so the penalty was 0 and a stack half of whose rubric never ran scored 100 / A / `valid: true` with every count zero — byte-for-byte the verdict a genuinely clean stack gets, on both faces (`os lint --score` and the eval harness's `passed`). The throw is reachable on a schema-valid stack: a localized `label` on an app or on a view's `list` parses clean and makes the label-case rule throw. That rule's crash is a separate defect; this change stops the scorer publishing a clean verdict it did not earn. A crashed run is now carried by `lintError` (new optional field), by a synthetic `error` issue (`rubric/lint-crashed`) so `counts.errors` and `valid` report it, and by `score` 0 / grade `F`. The schema verdict is untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D47qPfEWVPmhguWgBZCi5N
📓 Docs Drift Check4 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run. What this run could not see
Coarse fallback — 22 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin f2e617487b26223b26a774c180680fb5864adf22 && git checkout f2e617487b26223b26a774c180680fb5864adf22
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 8c7b2394f27095995bc0d5b487cc0242319511c2 c4f2e7ae671b3e9044236cc4e79cc56f6eafbadd && git checkout -B drift-repro 8c7b2394f27095995bc0d5b487cc0242319511c2 && git merge --no-ff c4f2e7ae671b3e9044236cc4e79cc56f6eafbadd
node scripts/docs-audit/affected-docs.mjs --json 8c7b2394f27095995bc0d5b487cc0242319511c2 |
os-litant
left a comment
There was a problem hiding this comment.
Contract review (clause ②) — PASS
Reviewed at head c4f2e7ae671b3e9044236cc4e79cc56f6eafbadd (merge-base with origin/main 59953d5a3). Posted as a COMMENT, not an approving review: GitHub refuses APPROVE on a PR authored under this same account, and agent seats do not submit approving reviews in any case. This is the in-seat contract review the card was dispatched under; the label stroke and landing are the seat's own and are not taken here.
Independence pair (C4):
Implemented-by:branchclaude/issue-15658-score-lint-crash-visible(subagent dev)Reviewed-by:session_01D47qPfEWVPmhguWgBZCi5N
Governed surfaces touched: none (.changeset/…, packages/cli/src/lint/score.ts, packages/cli/test/score-lint-crash.test.ts). metadata-eval.ts untouched, content/docs/releases/ untouched, no packages/spec/src/** file touched.
Clause ② — re-judged from the delivered diff, per limb
Mechanical limb — YES, and narrower than declared. packages/cli/src/index.ts re-exports commands only; score.ts is not on the package's . export and the exports map has no deep-import subpath ⇒ the mechanical limb's "new exported symbol on an index" leg does not apply, and the mechanical YES rests solely on the os lint --eval --json wire key. That key is real: driven through a dist built from this head, results[].score.lintError is an own property on every crashed case (5 of 5 lines on the crash run, 0 on the clean control), and the new rule value rubric/lint-crashed rides the same payload in results[].score.issues[]. LINT_CRASHED_RULE and MetadataScore are not npm-public TypeScript API. The packages/spec/src/** path leg is correctly declared not applicable.
Conformance limb — YES, re-derived. The input class is "a schema-valid stack on which lintConfig throws"; the concrete members are apps[].label and views[].list.label carrying an inline locale map. Same process, same input, OLD (merge-base) scorer vs NEW scorer:
| score | grade | valid | counts.errors | lintError | issues[].rule | |
|---|---|---|---|---|---|---|
OLD, apps[].label = {en, zh-CN} |
100 | A | true | 0 | — | [] |
| NEW, same input | 0 | F | false | 1 | set | rubric/lint-crashed |
NEW, views[].list.label = {…} |
0 | F | false | 1 | set | rubric/lint-crashed |
| OLD and NEW, string label (control) | 100 | A | true | 0 | — | [] |
The unchanged harness (metadata-eval.ts, passed reads counts.errors) fails the case at minScore 75 and at 0 (passed:false, ok:false, meanScore:0); the clean control passes at both. On the wire (os lint --eval --json --generator): crash generator EXIT=1, ok:false, 0/5 passed, meanScore 0; clean-generator control EXIT=0, ok:true, 5/5, meanScore 100. The wire-level before (exit 0) is derived, not driven by me: OLD scorer returns 100/A/valid:true with every count zero, passed (unchanged) is therefore true, ok true, and lint.ts exits 1 only on !report.ok — the dev's ablated-build table is taken on trust for that one row.
minor vs major — explicit verdict: minor is correct; not major; and no BREAKING banner or ADR-0087 disposition is owed
I did not take this from the green gate. The changeset gate checks the declaration's shape; the ADR-0087 gate is, by its own header, blind to omission — so "the gate confirms this declares no breaking change" is a tautology, not evidence. The ruling below stands on the merits.
- The level axis. Under the maintainer's 2026-09-04 WHICH LEVEL ruling (
pr-automation.yml, Check Changeset step) and the header ofcheck-changeset-no-major.mjs, the bump level does not carry breaking-ness during the launch window:majoris refused, and breaking-ness is carried by the BREAKING banner plus the ADR-0087 disposition. A new accepted key and a new accepted value on a published payload set the floor atminor, and afix(type cannot lower it.minoris exactly what the ruling prescribes;majorwould be wrong on this axis regardless of the next question. - The breaking axis — is a BREAKING banner owed? No.
- The repo's own definition (AGENTS.md rule 3): a breaking changeset "removes or renames anything an author can write (a spec key, an export, a config field)". This PR removes and renames nothing. Every existing key keeps its shape and meaning; the additions are optional and additive; a consumer that parsed the old payload parses the new one.
- The behaviour flip is confined to an input class whose prior output was a defect against the rubric's documented contract —
score.tssays a stack is good "exactly when it (a) parses and (b) is clean under the data-model lint rules"; a stack whose lint never ran was never established as (b). The 100 / A / exit 0 was unearned, not promised. Correcting an incorrect verdict for one input class is a fix, and this repo has twice shipped the same-family correction that way: #15576 and #15659 both flipped a published number from 100 to 0 on an input class and landed aspatchwith no banner. - The same input already failed the tool's primary face before this PR:
os lintandos lint --scoreexit 1 on it, because the command-levellintConfigcall (lint.ts:583) runs beforescoreMetadata(:613). No consumer could hold a consistent green on this stack across the tool; only the eval face disagreed with the rest of it. Aligning it narrows no contract. - There is nothing to migrate: an ADR-0087 disposition presupposes an author-side rewrite, and an author who meets
rubric/lint-crashedhas hit a linter bug (#15880), not a metadata shape they must change. - What I do acknowledge: a CI that ran
os lint --eval --generatoragainst a generator emitting localized app or view labels goes from green to red. That is the correct verdict for that generator under the rubric and is the pass-flip #15658 was filed to close. It deserves plain words in the changeset body — which it has ("a stack half of whose rubric never ran came back as 100 / A") — not a banner whose meaning here is "you must rewrite something".
I18nLabelSchema — re-derived, admits the locale map
I18nLabelSchema = z.union([z.string(), InlineLocaleMapSchema]) (packages/spec/src/ui/i18n.zod.ts:247); InlineLocaleMapSchema = z.record(<key matching INLINE_LOCALE_KEY>, z.string()), and INLINE_LOCALE_KEY = /^(?!(?:key|defaultValue)$)(default|[A-Za-z]{2,3}(-[A-Za-z0-9]{2,8})*)$/ admits en and zh-CN. AppSchema.label: I18nLabelSchema (app.zod.ts:1291); ObjectStackDefinitionSchema.apps: z.array(AppSchema).optional() (stack.zod.ts:333). Executed at this head: safeParse success on apps[].label = { en, 'zh-CN' } and on views[].list.label = {…}; control on the schema axis: objects[].label is z.string() and the same parser rejects the map (objects.0.label: Invalid input: expected string, received object) — so the pass is not a parser that admits everything. Then lintConfig throws TypeError: Cannot read properties of undefined (reading 'toUpperCase') on both — checkLabelCase (commands/lint.ts:101-102) indexes label[0] on an object; control with a string label returns 0 issues and does not throw. Reachability (world i) is established; the "delete the catch" branch of the card is closed.
On the sweep's control. The 241 schema-INVALID crashers discriminate on the detector axis — they show the harness observes a throw from lintConfig at all, so "2 hits in 3,064" is not an always-zero detector. They do not discriminate on the filter axis (that "schema-valid" survivors really passed safeParse); that axis is covered by the objects[].label rejection, which I reproduced. The sweep's counts themselves are NOT MEASURED here (the sweep script is not in the PR); the card's question does not depend on them — one driven input settles reachability, and I reproduced both hits.
The three carriers — three readers, not three spellings
- Synthetic
errorissue — the only carrier the unchanged harness reads (passed⇐counts.errors === 0), the only onevalidreads, and what the human--evalline prints (e.g. rule: message). Measured: at--eval-min 0the case still fails only because of it. score: 0/F— the only carriermeanScorereads (it sumsscore.scorealone) and what the human per-case line prints; without it a crashed case contributes 100 to the mean, which re-opens #15659's exact defect. It is also the only contentos lint --score --jsonpublishes ({score, grade},lint.ts:627) — on the current input class that face never reaches the scorer, but the scorer's contract is for any throw from any rule.lintError— the typed cause, and the only carrier that distinguishes "the linter reported an error" from "the linter never ran" without matching a rule-id string.
Sound. One acknowledged cost: counts.errors now includes a non-lint event; acceptable, since the rule id and lintError disambiguate and a separate count would widen the counts shape.
lintError optional — right design, wrong stated reason. Right on its own merits: (i) generationError?: string on the same report's case result is already the present-when-set shape, so lintError? matches the idiom beside it; (ii) a required field would force unscorableScore() to pick a value for a case whose linter never ran for an upstream reason — undefined there would assert "ran clean", which is false; absent is the honest value; (iii) the docblock's "absent means it ran" is the semantics. The PR's stated reason — that a required field would have forced an edit into a file another card is fenced to — is not a reason a published type should take a shape; had required been right, the fence would have had to yield. It did not have to, because optional is right anyway. Recorded; no change requested.
Scope fence — leaving the crash to #15880 is right
On stronger grounds than the one stated. The scorer's contract is for any throw from any rule, and the test pins it through a mocked linter, so this PR's guarantee does not depend on #15880's fate; the crash's repair is a product/contract choice (guard on typeof label === 'string', or resolve the default-locale entry — and the objects[].label vs apps[].label asymmetry) that the issue correctly routes to triage; and no user is left worse off — os lint already exited 1 on the input before this PR, and the eval face now agrees with it. "Repairing it would delete the input that proves reachability" is the weakest of the three reasons (the suite does not rely on that input). Not a blocker.
Measurements (exit codes captured before any pipe; worktree at this head, git status clean)
| leg | result |
|---|---|
vitest run — score-lint-crash, score, metadata-eval |
EXIT=0, 3 files / 31 tests passed |
same-process OLD-vs-NEW probe (source via tsx; OLD = merge-base blob of score.ts) |
table above; controls green on both axes |
dist carries the change |
rubric/lint-crashed ×1 in dist/lint/score.js, ×0 in the merge-base blob; on-disk score.ts blob = HEAD blob eef1cc1b |
os lint --eval --json --generator via bin/run.js (dist) |
crash EXIT=1 / clean control EXIT=0; details above |
check-changeset-no-major / check-adr-0087-registration / check-empty-changeset (--base origin/main) |
EXIT=0 / 0 / 0 (re-run here, not read from CI) |
| CI at head | Test Core (6/6), TypeScript Type Check, Build Core, Check Changeset, Dogfood gates: all success |
Resolution path, stated: the unit probe and the tests reach score.ts through source; the wire runs reach it through a dist built from this head (JS-only, OS_SKIP_DTS=1, lint-command closure of 16 packages plus @objectstack/cli). That partial build leaves oclif ModuleLoadError warnings on stderr for unrelated commands (cloud:*, data:*, …); none for lint, and stdout is well-formed JSON.
NOT MEASURED (neither pass nor red): the mutation sweep's counts; the dev's ablation (6 RED / 25 GREEN prediction) — not re-run; the wire-level before row (derived from the OLD scorer's output and the unchanged passed/exit logic, not driven); pnpm lint and pnpm typecheck locally — not re-run (CI's TypeScript Type Check is green at this head; the local cli build ran --noCheck); DTS emission — skipped in the local build.
Non-blocking notes
- The PR body cites the ADR-0087 gate as confirming non-breaking-ness; that gate cannot confirm it (blind to omission by design). The reasoning above is what carries the ruling; the outcome is unchanged.
- An
Error('')thrown by a rule yieldslintError: "", which a truthiness-reading consumer would miss; the other two carriers still hold. Exotic; not worth a change.
Generated by Claude Code
Fixes #15658
The card's open question, settled by measurement first
The card refused to pre-answer its own question and was right to: is the throw reachable at all, on a value that already survived
normalizeStackInputandObjectStackDefinitionSchema.safeParse? If not, the honest repair was to delete thecatch, not widen the report.World (i). The throw is REACHABLE, and not exotically. A driven input:
safeParsereturnssuccess: true—apps[].labelisI18nLabelSchema, a union ofz.string()and an inline locale map, and the map form is the platform's own way to write a localized label.lintConfigthen throwsTypeError: Cannot read properties of undefined (reading 'toUpperCase'), becausecheckLabelCaseindexes the label as a string. Onorigin/mainthe scorer swallowed that intoissues: []and returned:{"score":100,"grade":"A","valid":true,"counts":{"schemaErrors":0,"errors":0,"warnings":0,"suggestions":0}}views[].list.labelwith the same locale map does it too.How the input was found, with its control. A single-value mutation sweep over
examples/app-todo's normalized config — every path, eight hostile values, 15,728 mutations — kept 3,064 schema-valid, of which exactly two crashlintConfig(the two above). Positive control in the same run: 241 schema-INVALID mutations also crash it, so the detector fires and the two hits are not an artefact of an empty scan. A second sweep over the four bundled eval-corpus fixtures (2,745 mutations, 241 schema-valid survivors, control 314) found none — those fixtures declare noappsand noviews.So the repair is the one this repo has chosen twice before (#10653, #10123): make the failure visible in the output, keep the guard.
checkLabelCaseassuming a string is a bug inpackages/cli/src/commands/lint.ts; repairing it in this PR would delete the very input that proves this path reachable, and it is a different defect class. Filed with its measurement as #15880.Both published faces, measured — and one correction to the card
The card lists two faces. Driven end to end against a CLI built from this tree, they behave differently, and the difference is worth recording:
os lint --score— already loud, before and after. The command callslintConfig(normalized, { sduiManifest })at its own step, beforescoreMetadata, so the same throw kills the run first:The silent 100 the card predicts for this face is not reachable through this input: the only difference between the two calls is
sduiManifest, and the command's call is the superset. That is a correction to the card's blast-radius bullet 1, not to its verdict.os lint --eval— the face where a failure became a PASS. It reachesscoreMetadatadirectly, with no earlier lint call. Same generator, same tree, before and after (before = this branch with thecatchbody ablated back toissues = [], rebuilt, so thedist/the CLI runs really carried the old behaviour):okpassedfailedmeanScoretrueA/valid: true, counts all 0falseF/valid: false,errors: 1,lintErrorsetA generator whose every output crashed the linter earned a green eval and exit 0. That is the pass-flip the card graded correctly.
What changed
packages/cli/src/lint/score.tsonly. Thecatchno longer discards; it records, in every carrier a consumer might read, because reading any one of them has to be enough:lintError?: string— new optional field onMetadataScore, the thrown message. Set only when the linter could not run; absent when it ran and reported errors, which is a lint verdict rather than a missing one. Optional on purpose:unscorableScore()inmetadata-eval.tsconstructs aMetadataScoreliteral, and a required field would have forced an edit into a file another card is fenced to.errorissue,rule: 'rubric/lint-crashed'(exported asLINT_CRASHED_RULE) — soissues,counts.errorsandvalidcarry it. This is what makes the eval harness fail the case with no edit tometadata-eval.ts: itspassedreadscounts.errors, and would never have seen a new field. It is also what still fails the case at--eval-min 0, where the score alone stops discriminating — pinned by its own test.score: 0/ gradeF— the only channelos lint --score --jsonpublishes ({ score, grade }), and the same refusalunscorableScore()already gives an eval case there was nothing to judge, for the same stated reason: a clean number nothing earned is the worst possible output.The schema half is untouched and still reported:
schemaErrorsandcounts.schemaErrorssay exactly what the parse found. That was the defensible half of the original comment's intent, and it survives.Why this shape (the four axes)
unscorableScore()returns 0 /F/valid: falseand its docblock says why — ⛔ deliberately notscoreMetadata({}), because "a clean number nothing earned" is the failure mode. Reusing that verdict for "the rubric did not run" keeps one answer in the codebase instead of two. Adding a lower number without a flag would have been the alternative the card warns about (one wrong number for another); adding a flag without the number would have leftos lint --score --json, which publishes nothing butscoreandgrade, exactly as misleading as before.apps[].labelaccepting a locale map is a shipped authoring feature, and the crash it causes is reachable from a schema-valid stack today.Ablation
Prediction written before the run (
catchbody reverted toissues = [], marker injected, nothing else moved): 6 RED by name, 3 GREEN in the same file plus all ofscore.test.tsandmetadata-eval.test.ts, and the two schema assertions insidekeeps the schema verdictstaying green because the parse half is untouched.Mutation proven on disk before the run — removed text
rule: LINT_CRASHED_RULE1 → 0, injected markerABLATION-156580 → 1, blob hasheef1cc1b→4c74b941. Restore under anEXIT INT TERMtrap, proven after: disk hasheef1cc1bequals the HEAD blob, andgit diff HEADis empty. The test reaches the subject through source (test/score-lint-crash.test.tsimports../src/lint/score.js), andpackages/cli/distheld 0 entries at mutation time; for the CLI-face table above, wheredistIS the path, the ablated build was verified to carry the marker (1) and the restored build to have lost it (0).Result — exactly the prediction:
with the six failures being the six named ones, and
keeps the schema verdictfailing only onexpected undefined to be 'boom'at its third assertion.Verification — exit codes captured before any pipe
At
c4f2e7ae671, working tree clean.pnpm --filter '@objectstack/cli^...' build—VERDICT command-exit 0(the dependency closure;@objectstack/specand@objectstack/lintresolve throughexportstodist/).pnpm --filter @objectstack/cli exec vitest run --maxWorkers=2overscore-lint-crash,score,metadata-eval,lint-eval-json-unscorable-stack.e2e,lint-view-label,lint-namespace-prefix— 6 files / 63 tests passed,VERDICT command-exit 0.pnpm --filter @objectstack/cli typecheck—VERDICT command-exit 0;check:test-typecheck: OK. Both changed files are really in that program:tsc -p packages/cli/tsconfig.test.json --listFilesnamesscore-lint-crash.test.ts(1) andsrc/lint/score.ts(1), so this is a measurement and not a NOT MEASURED.pnpm lint— the repo-wideeslint . --no-inline-config, not a narrowed run:VERDICT command-exit 0, 77s.scripts/pm/dispatch-gates.mjs(derived from the delivered diff at this commit,--repo objectstack-ai/objectstackasserted), each captured ascmd > log 2>&1; EXIT=$?:nul-bytes,verify-stand-in,single-claim-paths,cli-test-child-env,cross-package-test-inputs,test-source-alias,type-source-resolution,changeset-gate-self-tests,check-changeset-no-major,check-empty-changeset,check-adr-0087-registration,check-comment-mask-adoption,objectql-double-limit,where-matcher,error-code-casing,doc-authoring,objectui-changeset,check-keyed-text-bounds— all EXIT=0, quoting their own verdict lines, e.g.check-nul-bytes: OK (scanned 7655 text file(s) …),✓ This diff introduces no 'major' bump.,check-test-source-alias OK — 72 packages with tests scanned. The remaining families the derivation lists arepackages/**-glob matches CI runs in full; that half is CI's, by the dispatch's own rule.Clause ② (契约复审) — declared from the delivered diff
Mechanical / path limb — YES.
lintErroris a new key on a published payload:os lint --eval --jsonemits the whole report, andresults[].scoreis theMetadataScoreobject. The key is in the JSON above, measured, not inferred. Thepackages/spec/src/**path leg does not apply — no spec file is touched.Non-mechanizable conformance limb — YES, and driven. The before/after table is exactly the shape the doctrine names: an input class that passed on a shipped face (
ok: true, exit 0) now fails (ok: false, exit 1). Not a re-reading of a borderline case — a measured verdict flip.Bump:
minor, per the Check Changeset step's WHICH LEVEL section (scripts/check-changeset-no-major.mjs:54) — a new accepted key on a published surface takes at leastminor, and the commit type may raise but never lower it. Notmajor: the gate refuses it, andcheck-adr-0087-registrationconfirms this diff declares no breaking changeset.Scope
#15578is a separate card and its repair has landed onmainalready; nothing here touchesmetadata-eval.ts. The label-case crash that makes this path reachable is filed as #15880 and is not repaired here.🤖 Generated with Claude Code
https://claude.ai/code/session_01D47qPfEWVPmhguWgBZCi5N
Generated by Claude Code